ON THE POLYNOMIAL AUTOMORPHISMS OF A 

GROUP 



GERARD ENDIMIONI 

Abstract. Let A(G) denote the automorphism group of a group 
G. A polynomial automorphism of G is an automorphism of the 
form x i— > (vi 1 x ei vi) . . . (v^ l 1 x em v m ). We prove that if G is nilpo- 
tent (resp. metabelian), then so is the subgroup of A(G) generated 
by all polynomial automorphisms. 



1. Introduction and main results 

Let G be a group. We shall write A(G) for the automorphism group 
of G. According to Schweigert [10], we say that an element / G A(C7) 
is a polynomial automorphism of G if there exist integers E\, . . . , e m G Z 
and elements ito, ■ ■ ■ , u m G C7 such that 

f(x) = UqX^Ux . . . u m -ix em u m 

for all x G G. Since /(l) = 1, it is easy to see that /(x) can be 
expressed as a 'product' of inner automorphisms, that is 

f(x) = (v^x^Vi) . . . (v~ l x fm v m ). 

We shall write Po{G) for the set of polynomial automorphisms of G. 
Actually, Schweigert defines a polynomial automorphism in the context 
of finite groups. In particular, in this context, the set Pq(G) is clearly 
a subgroup of A(C7). On the other hand, this is not necessarily the 
case when G is infinite. For instance, in the additive group of rational 
numbers, the set of polynomial automorphisms forms a monoid with 
respect to the operation of functional composition, which is isomorphic 
to the multiplicative monoid Z \ {0}. 
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In this paper, we shall consider the subgroup P(G) = (P (GQ) of 
A(G), generated by all polynomial automorphisms of G. Hence Pq{G) = 
P(G) when G is finite, but for example P(G) is distinct from Pq(G) 
when G is the additive group of rational numbers (note that P(G) = 
A(G) in this last case). 

It is easy to verify that Pq{G) is a normal subset of A(G). Thus 
P(G) is a normal subgroup of A(G); in addition, we have 

P\G) < P{G) < A(G), 

where 1(G) is the group of inner automorphisms of G. Also P(G) 
contains the group of invertible elements of the monoid Pq(G). It is 
worth noting that there exist finite groups G such that the quotient 
P(G)/l(G) is not soluble [7]. 

If G is abelian, each polynomial automorphism is of the form x i— ► x € , 
and so P(G) is abelian. When G is a finite nilpotent group of class 
k > 2, it is proved in p[] that P(G) is nilpotent of class k — 1 (see also 
[TU1 Satz 3.5]). We show here that this result remains true when G is 
infinite. 

Theorem 1.1. Let G be a nilpotent group of class k > 2. Then P(G) 
is nilpotent of class k — 1 . 

Notice that conversely, if P(G) is nilpotent, then so is G since P(G) 
contains the group of inner automorphisms. 

When G is metabelian, it seems that nothing is known about P{G), 
even in the context of finite groups. In this paper, we shall prove the 
following. 

Theorem 1.2. Let G be a metabelian group. Then the group P(G) is 
itself metabelian. 

In Section 3, we shall interpret a result of C. K. Gupta as a very 
particular case of this theorem (see Corollary 3.1 below). 

2. Proofs 

As usual, in a group G, the commutator of two elements x, y is de- 
fined by [x,y] = x~ x y~ x xy. Instead of we shall write [x,y,z\. 
We denote by [G, G] the derived subgroup of G. 
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Lemma 2.1. Let f,g be two functions over a group G, respectively 
defined by the relations 

f(x) = (vi 1 x ei v 1 )...(v- 1 x em v m ), 
g(x) = (w^ 1 x Vl Wi) . . . (w^x^Wn) 

(we do not suppose that f and g are automorphisms) . Let t be an 
element of G such that any two conjugates of t commute. Then we 
have the relation 

m n 
i=l j=\ 

(notice that in this product, the order of the factors is of no conse- 
quence). 

Proof. Using the fact that any two conjugates of t commute, we can 
write 

m / n \ ei 

f(g(t)) = U v ^ \U w 7 HriJw A v > 
i=i \j=i J 

m n 

i=l j=l 
m n 

= nn r '" j [ /f '" , - u '^]- 

i=l j=l 

We conclude thanks to the relation [x, yz] = [x, z] [x, y] [x, y,z]. □ 

In a nilpotent group G of class < 2, two conjugates of any element 
t G G commute. Therefore, as an immediate consequence of Lemma 
2.1, we observe that any two polynomial automorphisms of G commute. 
Since these automorphisms generate P(G), we obtain: 

Corollary 2.1. If G is a nilpotent group of class < 2, then P(G) is 
abelian. 

We are now ready to prove our first theorem. 

Proof of Theorem 1.1. Since P(G) contains 1(G) (which is nilpotent of 
class k — 1 exactly), it suffices to show that P(G) is nilpotent of class 
at most k — 1. We argue by induction on the nilpotency class k of 
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G. The case k = 2 follows from Corollary 2.1. Now suppose that our 
theorem is proved for an integer k > 2 and consider a nilpotent group 
G of class k + 1. Denote by C(^) the centre of G. One can define a 
homomorphism B : P(G) —>■ A(G/((G)), where for each / G P(G), 
6(/) is the automorphism induced by / in G/((G). Clearly, if / is a 
polynomial automorphism of G, then 0(/) is a polynomial automor- 
phism of G/((G). Hence 0(P(G)) is a subgroup of P(G/((G)), and so, 
by induction, is nilpotent of class at most k — 1. Since B(P(G)) and 
P(G)/kerB are isomorphic, it suffices to show that kerB is included 
in the centre of P(G) and the theorem is proved. For that, consider 
an element g G kerB and put w(x) = x~ 1 g(x) for any x in G. Thus 
g(x) = xw(x) and w(x) belongs to C(C) f° r & U x E G. Notice that w 
defines a homomorphism of G into C(C) since 

iw(a^/) = = y' 1 w(x)g(y) = w(x)w(y). 

In order to show that g belongs to the centre of P(G), it suffices to 
verify that g commutes with any polynomial automorphism / of G. 
Suppose that / is defined by the relation 

f(x) = (v^x ei vi) . . . (y^x tm v m ). 

We have easily 

f(g(x)) = f(xw(x)) = f(x)f(w(x)) = f(x)w(x) e , 

where e = e\ + • • • + e m . In the same way, by using the fact that w is 
a homomorphism, we can write 

g(f(x)) = f(x)w(f(x)) 

= f(x)(w(v 1 )' 1 w(x) £l w(v 1 )) . . . (w(v m )~ 1 w(xY m w(v m )), 

whence g(f(x)) = f(x)w(x) e . Thus g and / commute, as required, and 
the result follows. □ 

Now we undertake the proof of our second theorem. First we need 
the following result, which is well known and easy to prove (see for 
example [H Lemma 34.51] or 0, Part 2, p. 64]). 

Lemma 2.2. In a metabelian group G, ift is an element of the derived 
subgroup [G,G\, we have the relation [t,x,y] = [t,y,x] for allx,y G G. 
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We arrive to the key lemma in the proof of Theorem 1.2. This 
lemma shows that when G is metabelian, any element h G [P(G), P(G)] 
operates trivially on [G, G] and on G/[G, G\. 

Lemma 2.3. Let G be a metabelian group. Suppose that h is an ele- 
ment of the derived subgroup [P(G),P(G)}. Then 

(i) h{t) = t for allte [G, G\; 

(ii) x^h(x) belongs to [G,G] for all x G G. 

Proof, (i) Consider the homomorphism $ : P(G) — > A([G,G]) defined 
like this: for any / G P(G), $(/) is the restriction of / to [G, G}. We 
must show that ker <3> contains [P(G), P(G)]. For that, first notice that 
any two conjugates of t G [G, G] commute since G is metabelian. Now 
we apply Lemma 2.1. If / and g are polynomial automorphisms of G 
defined as in this lemma, we obtain the equalities 

m n 

i=l j=l 
m n 

i=l j=l 

and so, by Lemma 2.2, f(g(t)) = g(f(t)) for all t G [G, G]. It follows 
that [/, g] belongs to ker <3>. In other words, the images of / and g in 
P(G)/ker$ commute. Since P(G)/ker$ is generated by the images 
of the polynomial automorphisms, this quotient is abelian. It follows 
that ker$ contains [P(G), P(G)], as desired. 

(ii) Here, we consider the homomorphism ^ : P(G) — > A(G/[G, G]), 
where for any / G P(G), ^(/) is the automorphism induced in G/[G, G] 
by /. Since a polynomial automorphism of G induces in G/[G,G] 
a polynomial automorphism of G/[G,G], ^(P(G)) is a subgroup of 
P(G/[G, G]). But P{G/[G, G]) is abelian (see for instance Corollary 2.1 
above) and *(P(G)) is isomorphic to P(G)/ker^. Hence P(G)/ker^ 
is abelian. Consequently, ker\l> contains [P(G), P(G)] and the result 
follows. □ 

Proof of Theorem 1.2. Let f,g be two elements of [P(G), P(G)]. For 
any x G G, put v(x) = x~ l f(x) and w(x) = x^g^x). By Lemma 2.3, 
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v(x) and w(x) belong to [G, G]. Applying again Lemma 2.3, we can 
write 

f(g(x) = f(xw(x)) = f(x)f(w(x)) = xv(x)w(x). 

In the same way, we have g(f(x)) = xw(x)v(x) = xv(x)w(x). It follows 
that / and g commute. Thus [P(G),P(G)] is abelian, and so P(G) is 
metabelian. □ 



3. IA-AUTOMORPHISMS OF TWO-GENERATOR METABELIAN GROUPS 

By way of illustration, we apply Theorem 1.2 to IA-automorphisms 
of a two-generator metabelian group. We recall that an automorphism 
of a group G is said to be an I A- automorphism if it induces the iden- 
tity automorphism on G/[G,G}. In a free metabelian group of rank 
2, each IA- automorphism is inner pQ, and so is a polynomial automor- 
phism. It turns out that in any two-generator metabelian group, each 
IA-automorphism is polynomial. This result is implicit in [2] with a 
different terminology. For convenience, we give a proof since this one 
is short and elementary. 

Proposition 3.1. Each IA-automorphism of a two-generator metabelian 
group is polynomial. 

To prove this proposition, we shall use the following result. 

Lemma 3.1. In a metabelian group G, each function tp of the form 

X I— > <f(x) = x[x, Vi] Vl . . . [x, V n ] Vn (Vi G G, Tji G Z) 

is an endomorphism. 

Proof. Thanks to the relation [xy, z] = y~ l [x, z]y[y, z], we get 

n 

ip(xy) = xy Yl (y^lx.v^yly.v^Y 1 . 
i=i 
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But since the derived subgroup of G is abelian, we can write 

n n 

ip(xy) = xy Y[ (y-^x^v^y) 



i=l 



= <p(x)(p(y), 



.i=l / / i=l 



as required. □ 

Proof of Proposition 3.1. Suppose that G is a two-generator metabelian 
group generated by a and b. If / is an IA-automorphism of C7, we have 
f(a) = av and f{b) = bw, where v and w belong to the derived sub- 
group [G, G\. Now notice that [G, G] is the normal closure of [a, b]. 
Therefore, [G, G] is generated by [a, b] and the elements of the form 
[a, b, u] , with u G G. Hence v and w can be written in the form 

n 

i=i 

n 

w = [a,bf Y[[a,b,Wi}^, 

i=i 

where a, (3, X±, . . . , A n , fj,i, . . . , /i„ are integers (possibly equal to 0). By 
using the relation [x,y,z] = [x, y]" 1 ^, z}' 1 ^, yz], we obtain 

n 

v = [a,b] a ~ x l[[a,v t ]~ Xl [a,bv t ] X % 

i=i 

n 

w = [a,b}^l[[a,w t ]-^[a,bw t r% 
i=i 

where A = Ai + • • • + A n and /i — /ii + ■ — h jj, n . Now put 

n 

ip(x) = x[x, b} a ~ x [x, a]"-* ]J[x, Vi ]- Xi [x, bv^x, WiY* [x, aw^. 

i=i 

By Lemma 3.1, ip is an endomorphism of G. Moreover, we have 

n 

<p(a) = a[a,b] a - x JJ[a, ^]" Al [a, 6^] Al = av = /(a) 



cS 



GERARD ENDIMIONI 



since [a, Wi) = [a,awi\. 
In the same way, we get 

n 

<p(b) = b^bf-^W^w^aWi}-^. 

i=i 

By using the identity [a, Wj] _1 [a, bwi] = [b, awj] _1 [6, wi\ (valid in any 
group), we obtain 

n 

<p(b) = b[a,bf-^l[[a,w i \-^[a,bw i r i = bw = f(b). 
i=i 

Thus f — (f and the proof is complete. □ 

We remark that Proposition 3.1 cannot be extended to three-generator 
metabelian groups. For example, in the free metabelian group of rank 
3 freely generated by a, b, c, consider the IA-automorphism / defined 
by f(a) = a, f '(b) = b and f(c) = c[a, b]. Suppose that / is polynomial. 
Since [a, b] = c _1 /(c), the commutator [a, b] would be in the normal 
closure of c, hence would be a product of conjugates of c ±x . Substi- 
tuting 1 for c in this expression gives then [a, b] = 1, a contradiction. 
Therefore / is an IA-automorphism which is not polynomial. 

As a consequence of Theorem 1.2 and Proposition 3.1, we obtain an 
alternative proof of a result due to C. K. Gupta [6] (see also [3]). 

Corollary 3.1 (|6J). In a two-generator metabelian group, the group 
of I A- automorphisms is metabelian. 

Let Md denote the free metabelian group of rank d. By a result of 
Bachmuth [I], if d > 3, the group of IA-automorphisms of contains 
a subgroup which is (absolutely) free of rank d. Thus Corollary 3.1 
fails in a (i-generator metabelian group when d > 3. Also Bachmuth's 
result shows once again that the group of IA-automorphisms of Md is 
not included in P(M^) (if d > 3), since P(M^) is metabelian. 

In conclusion we mention that the metabelian groups constitute an 
important source of polynomial endomorphisms and automorphisms. 
Indeed, by Lemma 3.1, each function of the form 

X \—*x[x, Vi] V1 . . . [x, Vn} 11 " (Vi G G, T]i G Z) 
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is an endomorphism in a metabelian group G. Besides, when G is 
metabelian and nilpotent, such an endomorphism is an automorphism 
since in a nilpotent group, every function of the form 

x i — > u x €1 U\ . . . u m -ix em u m (Ui G G, €i G Z) 

is a bijection if e± + ■ ■ ■ + e m = ±1 (see [5], Theorem 1]). 
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